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^his paper describes the instructional system 
operative at the department level at most American colleges and 
universities and indicates the function and relationship of 
development and evaluative activities within it. It suggests that 
many of the instructional systems within the university structure mav 
be legitimately character iied as closed systems; that one trait of 
such systems is their limited use of external and internal feedback 
messages; that educational evaluation, whether formative or 
summativo, must he characterized by tb° de V e loner as pari of a 
orocess that will lead to change: and that the initiation of this 
process occurs with the development of tpo cl ient - proh len statement. 

A heuristic model designed to assist the course developer who is 
emnloyinq an evaluative orocess within a closed, instructional svsto^ 
is then outlined to facilitate with this set of verbal 
generalizations the formation of f he substantive problem statement. 
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The instructional system that is operative at the department level 
of most American universities is notoriously insular and closed. My pur- 
pose today t9 to describe that system; to indicate the function and rela- 
tionship of development and evaluative activities within it; and to 
describe the primary factors that compose a heuristic uodel of instructional 
development that will have application to it. 

A system is an interrelated set of people* objects, or messages. 
Examples range from the broad* complicated ecological systems that 
describe fundamental relations in socle 4 / and nature, to the equally com- 
plicated cybernetic system of the human organism. In addition to their 
complexity and comprehensiveness* these systems may be differentiated 
through a consideration of their stability, structure and function, goal 
parameters, and feedback channels. 

1/hen I say that a system is closed, what 1 suggest is that the 
goal parameters that define its boundaries and the feedback channels that 
it contains, and that are supposed to connect it to other systems, are 
not responsive to super and ancillary systems surrounding it. 

By an instructional system I refer to a set of people, objects, 
and messages organized in structure and function to achieve a learning 
outcome. This concept subsumes several others that initiate in the 
cybernetir view of an individual's perceptive and introspective activity 
and range up through the tutorial, classroom, departmental and higher 
order supersystems. Its extension at any one of these levels would 
include varying distributions of general and specific system character- 
istics; such as, size, stability, specificity of function and goal para- 
meters together with the extent and efficiency of internal and external 
feedback channels. 

Viewed at the department level, for example, the instructional 
system that exists at many major universities includes numerous course 
level subsystems, and individual cybernetic systems. It is also laterally 
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connected to ancillary support systems. Moreover, it forms one component 
within the broader administrative and academic structures of the univer- 
sity. Consequently, to construct a series of activities that will achieve 
change in this instructional system and move it from closed to open is an 
extremely complicated task, to say the least. 

As I have suggested, one of the major characteristics of a closed 
instructional system is its limited use of external messages. The system 
that is stable and autonomous can afford to move without too much regard 
for the functions or messages of other ancillary or supersystems. Conse- 
quently, much of the project evaluation under ESEA, for example, has taken 
place as the result of a contractual relationship, a message to which the 
system had to respond. Contractual obligations require at least acquies- 
cence, but they do not require interest in evaluation at the project level. 
Unfortunately, even the authority and force of mandate are denied the devel- 
oper. As a result, where the outside evaluator may be able to deal from a 
position of contractual authority, the course developer must demonstrate 
th^ utility of his output to assure continuation of what must be a cooper- 
ative effort. This problem is clearly manifest at the course level of 
instruction within the university structure. 

Fortunately, one of the spin-offs that may be developing from the 
increasing rate of student participation in curricular matters may be 
the use of course evaluation devices sponsored by independent student 
organizations. These together with more traditional methods of course 
rating, such a9 the Course Evaluation Questionnaire used at the Univer- 
sity of Illinois, have begun to produce sufficient data pressure to 
cause the instructional system to attend to it, even if only to deny its 
utility. Yet in the denial itself, there is the recognition of a message 
received, and this event itself may become an opportunity for developers 
to achieve change, 

And change is the crux of the thing, but not for its own sake. 
Scriven's distinctions between formative and suircnative evaluation were 
helpful to the evaluator when he tried to distinguish clearly his roles 
and goals, but possibly, their application is considerably limited when 
they are viewed along the continuum of change or in cojrse development 
activity, for example. Other than as a chronological distinction, it is 
probably of small worth to separate the times when we are preparing instruc- 
tion from those in which we are summarily assessing it. Despite the fact 
that the questions that mark these activities may differ, we want to be 
able to legitimately characterize the focus of both formative and summative 
efforts as one that leads to change. Unfortunately, to characterize 
educational evaluation, as it is cocrcnonly practiced today, as a directed 
change process would be unwarranted, and possibly misleading. Clearly, 
there are still important questions to be considered as to whether change 
is a legitimate goal of evaluation in general. But I think it safe to 
say that at the development level, the implementation of evaluative 
methodologies must be such that the output from the process will accomodate 
a system aimed at instructional change. Yet, when viewed as a goal of 
university offices of course development, where they exist, the substantive 
relations that connect evaluation and change have not been delineated. 
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There are, however, numerous methodologies that have application 
to the problem of opening a closed system. One might, for example, 
attempt to affect the rewards structure through an incentive program 
emphasizing instruction and assume that this process will culminate in 
the redefinition of the system goal parameters. Again, there is much to 
be said for direct attempts to manipulate a pivotal structural component, 
or for gaining power, even for inciting the public verbalization of student 
dissatisfaction and satisfaction with instruction, where it may exist. 

It io my contention, as the result of considered experience, that 
ancillary systems designed to facilitate course development can capitalize 
on available feedback channels as primary and effective means for opening 
an instructional system. Practically speaking, what we are saying is that 
careful selection of information and its controlled insertion into the 
instructional system, when combined with over-riding concerns for the 
maintenance of a supportive relationship and credibility, will, over time, 
effect an alteration in the goa'i. parameters of that system. 

Given any developmental task> it may be a truism to point out that 
the type, quality, and amount of the information on the problem that will 
be required by different actors within the instructional system will vary 
according to their function in the system, and their personal predisposi- 
tions, For example, what may prove appropriate feedback for the course 
level instructor in regard to his content tests, may be irrelevant to the 
department head whose concerns about testing within the courses, although 
not antagonistic to the instructor's, may emphasize other factors. More- 
over, the selection of appropriate formate for the transfer of information 
vary from one element of the system to another. As a result, an accurate, 
well-documented report of a development process and recommendations, may 
function more as on artifact of the effort than as a message into a deci- 
sion system. 

It would seem that attention to the following questions should 
assist the developer in selecting the most appropriate feedback form 
and channel: 

J. What is the problem or task? 

2# What person or groups will need information for decisions on 
this problem? 

3, What kind of information will they most readily attend to? 

4, What feedback channels are presently available and open 
between the client and agent subsystems? 

5, When must information be available to the persons or groups 
so that it can be utilized? 

6, At what rate should infonnation be fed into the client system? 

7, What reportorial format will be mo9t reliable, efficient, and 
enticing? 
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The effective functioning of all of these question indicators is 
clearly dependent on the answer to the first; that is, the development 
of a comprehensive statement of the developmental task. This must con- 
sist of a clear indication of the forces to which the effort must be 
responsive and the goals it must achieve. Accomplishing this task with- 
in a developmental framework should result in a description of the client's 
perceived problem ; that is, his candid verbalization of the task. 

Experience suggests that the process of achieving and focusing this 
verbal statement may be a fumble-sorae one, and what is more, the outcomes, 
although they may be more precise than the client's initial queries, will 
necessarily be heavily value-laden since they will reflect his normative 
theories. For example, what began as a statement of concern about the 
validity of a supposed high rate of negative feedback from students in 
basic courses, may be refined to a concern over the number of students 
enrolling as majors within a department. This concern of the client will 
be necessarily juxtaposed against the values of the agent or developer, 
who in this case, may judge the nub of the problem to be one of an instruc- 
tional nature centered on the ability of staff to actively and affectively 
involve students in the learning process. 

The possible variance between a client's problem p erception and 
his ‘'real 11 problem , and again between the normative theories of tTie 
client and the agent, leads to a second concern. If, as Stufflebeam has 
suggested, one of the purposes of evaluation is to gather information for 
decision making, it would seem encumbent upon the evaluator to distinguish 
between information and data. Although these two terms differ in several 
respects, a primary one seems to be that information must be characterized 
by its relevance to a client's perceived problem while data may be descrip- 
tive of any phenomenon. The determination of what is relevant information 
may be based on a client's consideration of what he is willing to make a 
decision on, while the agent's concern may be more clearly formulated along 
traditional lines of scientific inquiry and research findings. 

The task of determining appropriate information is no where more 
clearly evident than it is when the role of the formal, external evaluator 
is compared to that of the internal developer who is employinging an eval- 
uative process as a means to assist change. In the case of the external 
evaluator, the client buys the judgments and valuations of the agent at 
contract. As a result, the determination of what is appropriate informa- 
tion is primarily that of the evaluator. On the other hand, in the case 
of the developer, the additional constraints of status, autonomy, threat, 
and credibility require an agent-client interaction that will be less 
prescriptive and less judgmental, initially at least, than that connecting 
the external evaluator and his client. 
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There are several important distinctions that separate evalua- 
tion from research and development. These have received excejlent 
attention in the literature* but I would like to emphasize that a prime 
difference between research and evaluation is the former r s concern with 
generalizability and the latter's emphasis on description, judgment, and 
the development of information for the management of conflict. Although 
it may be unwise to mistake research for evaluation, it seems equally 
misleading to split evaluation and course development. The relationship 
between the two is far more intimate than the word formative suggests. 
Where the task is to develop new programs or to improve existing ones, 
one aim of the developer should be to employ and to instill the evaluative 
process in order to describe and judge the program, and to establish 
within the program the procedures that will insure evaluation continuation 
after the formal relationship with the development agent has ended. 

Thus far we have suggested that many instructional systems within 
the university structure may be legitimately characterized as closed 
systems. In addition, I have pointed out that one trait of such systems 
is their limited use of external and internal feedback messages. More- 
over, we have argued that educational evaluation, be it formative o:r 
summative, must be characterized, by the developer, as part of a process 
that will lead to change. We have also suggested that the initiation of 
this process occurs with the develojxnent of the client-problem statement, 
and we have sketched several pivotaT aspects of this task. 

1 would now like to outline briefly a heuristic model designed 
to assist the course developer who is employing an evaluative process 
within a closed, instructional system. When I speak of a heuristic 
model, I am referring to a set of verbal generalizations that will facil- 
itate substantive problem statement. 

Conceptually, the relationship between a development agent and 
client locates within the larger boundary of a general system, which, 
for example, might be thought of as a university structure. The client 
and agent are representatives of their own subsystems within the larger 
boundary. (See Figure 1.) The genesis of a development task, or what 
we have termed a perceived problem, occurs within the client system as 
the result of a malfunction. 

Although destruction of the equilibrium within the client system 
may take many forms, all such problems may be initially viewed as some 
type of blockage that is interferring with the functioning of the system. 
To apply some of Stake's terminology, we might say that a perceived 
problem is actually a recognition on the part of a client that there is 
a lack of congruence among the intents, observations, and standards by 
which he judges and makes decisions. 

This is no strange phenomenon in instructional settings. We all 
have experienced the situation in which we have had expectations that 
were not fulfilled to a desired level. A good question to ask here is 
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FIGURE 1 



Initial Contact and Linkage Between an 
Instructional and a Development 
System 
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under v;hat conditions does this dissatisfaction manifest itself to an 
outside agent, in this case to a course development agent? An applicable 
construct for describing these conditions is range of tolerance . Just as 
individuals utilize defense mechanisms that help them sustain their self- 
image, systems have built-in adaptive devices and shunts that allow them 
to absorb negative feedback and to initiate corrective action designed to 
restore equilibrium. As long as malfunction does not exceed the accustomed 
limits of extremity, the range of tolerance, the individual and system will 
contain the dissonance. It is at the point where dissonance has exceeded 
the established range of tolerance that the probability of messages to an 
ancillary agency is greatest. 

One aspect of a closed system that challenges the course developer 
is its characterization by an implicit and traditional set of standards, 
so vague and so rooted that the system rarely requires an adjustment for 
congruence, Amitai Etzioni has suggested that it may be one of the roles 
of the development system to "define for the organization successively 
higher levels of satisfaction, by raising the standards of what [has been 
traditionally] considered f all right, 1 " (Page 31,) In one sense, then, 

I am in favor of dissonance. 

Once the initial contact between client and agent systems has 
occurred, the process of categorizing the problem begins. Simultaneously 
with the categorization of the problem, and partly as a result of the 
inter-relatedness of the systems’ components, the client and agent 
systems become wedded in a stronger set of communication channels and 
networks as the development process continues. (See Figure 2.) It is at 
this point that my earlier remarks regarding feedback, type, format, and 
capacity take on their real significance. The developer may begin with a 
closed system, but cncc having achieved entrance into the message circuit 
of that system, he must be extremely sensitive to the nature of ard the 
dissemination of the information he sends into it. 

The mediating factors that connect the perceived and the categorized 
problem are located in client-agent interaction. Both actors represent a 
network of system goals and personal predispositions. (See Figure 3.) 
Moreover, when the actors are heavily imbued with the system orientation, 
the movement from perception of problem to categorization of problem may 
bo very rapid, sincu the client will more than likely be seeking some 
procedural explanation or support. Such a case would be represented by a 
request to assist in the conversion of a hand scored test to o machine 
scored test. It may also be this system element in the client-agent inter- 
face that predominates when the relationship has occurred as the result of 
contractual obligations. 

On the other hand, where the initiation of the client-agent rela- 
tionship hak occurred os the result of a recognition of poor congruence, 
and where support, acceptance, and utility are the criteria of continu- 
ance, the personal factor may predominate. In cases of strong individual 
dissatisfaction with instructional outcomes or persoral feelings of in- 
adequacy, for example, the development of the categorized problem will be 
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significantly influenced by the personal aspect of this relationship. 
Moreover, I would like to suggest that in development activities what* 
many times begins as a formal relationship between client and agent, 
frequently evolves into a highly personal interaction. It is within 
these channels that additional complaints, reservations, and standards 
held by the client may be revealed. 

The outcomes of the interaction of these two primary mediating 
elements is the categorization of the real problem. When I say that the 
problem has been categorized, I mean that it has been located within an 
analytical framework that will facilitate the identification of broad 
problem areas and will also allow early specification of problem subsets, 
if there be any. Although designed for somewhat different purposes, 

Stake's classification system, developed in M The Countenance of Educa- 
tional Evaluation 1 ' is generally applicable in this endeavor. Without 
developing, at this time, the relationships that define and bind this 
categorization, a brief example may help you to conceptualize this 
operation. (See Figure 4.) 

Each problem may be seen as composed of a lack of congruence 
between a column of client intents, some client observations on the 
attainment of these intents, and a series of implicit or explicit 
standards by which judgments have been made. In addition, each problem 
may be viewed in another dimension through a consideration of the factors 
that may have led to the particular problem (antecedents), the processes 
or activities ^transactions) that are connected to it, and the outcomes 
that were intended and/or observed. This matrix may be ihoi as a specified 
set of dependent statements organized for a particular problem. Since 
problems are interdependent, I would have you envision any numbor of 
overlapping matrices, one for each problem. In one sense, what I'd like 
to suggest is that the complexity of a development task is, in part, 
dependent on the number of contingent problems that have to be peeled 
or pierced in order to achieve change. 

For example, what began not long ago, with an initial client phone 
contact revealed an instructor’s judgment that his tests were not really 
measuring the outcomes of his course. Some days later, a lengthy inter- 
view showed that he also had judged that his department's examinations were 
laden with recall items, that his department wa9 employing hand scoring 
testing procedures, and that key staff within the department did not see 
any utility in improving the department's testing program. Uhat we have 
here is not one testing problem but four, more or less complicated and 
highly interrelated problems. Locating each of them as an individual 
layer in the problem dimension of the matrix facilitated the specifica- 
tion of the real problem and also aided our realization that to initiate 
the transfer of the department's testing program to an automated system 
might be ill-advised, if key faculty actually did not see the utility of 
a testing development attempt to begin with. 

I would like to close on a note of encouragement and a suggestion 
of warning. I think there is encouragement in these remarks if only 
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The Categorization of Development Problems 
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because I have employed the model and it seens to have worked effectively 
as the beginning of an approach to instructional improvement in a univer- 
sity structure, Unfortunately, although there may be numerous methodolo- 
gies for instructional development, what we do not possess is a well- 
founded theory for selecting and ordering the scope and sequence of goals 
for a development system. Consequently, the involvement of such agencies 
frequently straps their available resources. What seems to occur is that 
entrance into the instructional system at any level, for any task, frequently 
leads to additional pursuits within the same course or department and forces 
an attenuation of resources. But possibly this is encouraging; possibly 
thi9 signals some success. 
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